{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build a RAG solution in Azure AI Search\n",
    "\n",
    "This notebook provides sample script for the indexing pipeline in [Build a RAG solution in Azure AI Search](https://learn.microsoft.com/azure/search/tutorial-rag-build-solution). If you need more information than the readme provides, you can refer to that article.\n",
    "\n",
    "Steps in this notebook include:\n",
    "\n",
    "- Set up the environment\n",
    "- Set up the Azure resources used in the pipeline\n",
    "- Create an index, data source, skillset, and indexer on Azure AI Search\n",
    "- Send a query to the search engine to check results\n",
    "- Send a query to an LLM to chat with your data\n",
    "- Revisit the index schema and query logic to improve relevance\n",
    "\n",
    "Sample data is a collection of PDF pages from the NASA's Earth Book that you load into Azure Storage and retrieve during indexing.\n",
    "\n",
    "This tutorial assumes embedding and chat models on Azure OpenAI so that you can use the integrated vectorization capabilities of Azure AI Search. You can use a different provider but you might need custom skills or a different approach for indexing and embedding your content."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "You need the following Azure resources to run all of the script in this notebook.\n",
    "\n",
    "- [Azure Storage](https://learn.microsoft.com/azure/storage/common/storage-account-create), general purpose account, used for providing the PDFs.\n",
    "\n",
    "- [Azure OpenAI](https://learn.microsoft.com/azure/ai-services/openai/how-to/create-resource) provides the embedding and chat models.\n",
    "\n",
    "- [Azure AI Services multiservice account](https://learn.microsoft.com/azure/ai-services/multi-service-resource), in the same region as Azure AI Search, used for recognizing location entities in the Earth Book.\n",
    "\n",
    "- [Azure AI Search](https://learn.microsoft.com/azure/search/search-create-service-portal), basic tier or higher is recommended. Choose the same region as Azure OpenAI and Azure AI multiservice.\n",
    "\n",
    "To meet the same-region requirement, start by reviewing the [regions for the embedding and chat models](https://learn.microsoft.com/azure/ai-services/openai/concepts/models#model-summary-table-and-region-availability) you want to use. Once you identify a region, confirm that Azure AI Search with AI services integration is available in the [same region](https://learn.microsoft.com/azure/search/search-region-support#azure-public-regions)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sign in to Azure\n",
    "\n",
    "You might not need this step, but if downstream connections fail with a 401 during indexer pipeline execution, it could be because you're using the wrong tenant or subscription. You can avoid this issue by signing in from the command line, explicitly setting the tenant ID and choosing the right subscription.\n",
    "\n",
    "This section assumes you have the [Azure CLI](https://learn.microsoft.com/cli/azure/authenticate-azure-cli-interactively).\n",
    "\n",
    "1. Open a command line prompt.\n",
    "\n",
    "1. Run this command to get the current tenant and subscription information: `az account show`\n",
    "\n",
    "1. If you have multiple subscriptions, specify the one that has Azure AI Search and Azure OpenAI: `az account set --subscription <PUT YOUR SUBSCRIPTION ID HERE>`\n",
    "\n",
    "1. If you have multiple tenants, you can list them: `az account tenant list`\n",
    "\n",
    "1. Sign in to Azure, specifying the tenant used for Azure AI Search and Azure OpenAI: `az login --tenant <PUT YOUR TENANT ID HERE> `\n",
    "\n",
    "You should now be logged in to Azure from your local device."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up Azure resources using the Azure portal\n",
    "\n",
    "We recommend using the Azure portal for setting up resources.\n",
    "\n",
    "You must be a subscription **Owner** or **User Access Administrator** to create roles. If you don't have permission to create roles, you can use API keys instead. If you're using keys, you can skip the steps that enable system assigned managed identities.\n",
    "\n",
    "1. Download the sample PDF files from [nasa-e-book/earth_book_2019_text_pages](https://github.com/Azure-Samples/azure-search-sample-data/tree/main/nasa-e-book/earth_book_2019_text_pages).\n",
    "\n",
    "1. Sign in to the [Azure portal](https://portal.azure.com).\n",
    "\n",
    "1. Make sure Azure AI Search, Azure OpenAI, and Azure AI multiservice resources are in the same region.\n",
    "\n",
    "### Configure Azure Storage\n",
    "\n",
    "1. On the Azure Storage left menu, select **Storage browser** > **Blob containers**, and then **Add container**.\n",
    "\n",
    "1. Name the container *nasa-ebooks-pdfs-all*.\n",
    "\n",
    "1. Upload the PDFs to the container.\n",
    "\n",
    "1. On the left menu, select **Settings** > **Identity** and turn on system assigned managed identity.\n",
    "\n",
    "### Configure Azure AI Search\n",
    "\n",
    "1. On the Azure AI Search left menu, select **Settings** > **Semantic ranker** and enable the free plan that authorizes 1,000 requests at no charge.\n",
    "\n",
    "1. On the left menu, select **Settings** > **Keys** and turn on role-based access control or \"both\".\n",
    "\n",
    "1. On the left menu, select **Settings** > **Identity** and turn on system assigned managed identity.\n",
    "\n",
    "### Configure Azure OpenAI\n",
    "\n",
    "Deploy the following models on Azure OpenAI:\n",
    "\n",
    "- text-embedding-3-large on Azure OpenAI for embeddings\n",
    "- gpt-4o on Azure OpenAI for chat completion\n",
    "\n",
    "You must have [**Cognitive Services OpenAI Contributor**]( /azure/ai-services/openai/how-to/role-based-access-control#cognitive-services-openai-contributor) or higher to deploy models in Azure OpenAI.\n",
    "\n",
    "1. Go to [Azure OpenAI Studio](https://oai.azure.com/).\n",
    "\n",
    "1. Select **Deployments** on the left menu.\n",
    "\n",
    "1. Select **Deploy model** > **Deploy base model**.\n",
    "\n",
    "1. Select **text-embedding-3-large** from the dropdown list and confirm the selection.\n",
    "\n",
    "1. Specify a deployment name. We recommend \"text-embedding-3-large\".\n",
    "\n",
    "1. Accept the defaults.\n",
    "\n",
    "1. Select **Deploy**.\n",
    "\n",
    "1. Repeat the previous steps for **gpt-4o**.\n",
    "\n",
    "Make a note of the model names and endpoint. Embedding skills and vectorizers assemble the full endpoint internally, so you only need the resource URI. For example, given `https://MY-FAKE-ACCOUNT.openai.azure.com/openai/deployments/text-embedding-3-large/embeddings?api-version=2024-06-01`, the endpoint you should provide in skill and vectorizer definitions is `https://MY-FAKE-ACCOUNT.openai.azure.com`.\n",
    "\n",
    "### Configure search engine role-based access to Azure Storage\n",
    "\n",
    "1. Sign in to the [Azure portal](https://portal.azure.com) and find your storage account.\n",
    "\n",
    "1. On the left menu, select **Access control (IAM)**.\n",
    "\n",
    "1. Add a role for **Storage Blob Data Reader**, assigned to the search service system-managed identity.\n",
    "\n",
    "### Configure search engine role-based access to Azure models\n",
    "\n",
    "Assign yourself *and* the search service identity permissions on Azure OpenAI. The code for this tutorial runs locally. Requests to Azure OpenAI originate from your system. Also, embedding requests and query reponses from the search engine are passed to Azure OpenAI. For these reasons, both you and the search service need permissions on Azure OpenAI.\n",
    "\n",
    "1. Sign in to the [Azure portal](https://portal.azure.com) and find your Azure OpenAI resource.\n",
    "\n",
    "1. On the left menu, select **Access control (IAM)**.\n",
    "\n",
    "1. Add a role for [**Cognitive Services OpenAI User**](/azure/ai-services/openai/how-to/role-based-access-control#cognitive-services-openai-userpermissions).\n",
    "\n",
    "1. Select **Managed identity** and then select **Members**. Find the system-managed identity for your search service in the dropdown list.\n",
    "\n",
    "1. Next, select **User, group, or service principal** and then select **Members**. Search for your user account and then select it from the dropdown list.\n",
    "\n",
    "1. Select **Review and Assign** to create the role assignments.\n",
    "\n",
    "This step concludes provisioning services in the Azure portal. Continuing to the next section, you switch to Visual Studio Code and a local environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a virtual environment in Visual Studio Code\n",
    "\n",
    "Create a virtual environment so that you can install the dependencies in isolation.\n",
    "\n",
    "1. In Visual Studio Code, open the folder containing tutorial-rag.ipynb.\n",
    "\n",
    "1. Press Ctrl-shift-P to open the command palette, search for \"Python: Create Environment\", and then select `Venv` to create a virtual environment in the current workspace.\n",
    "\n",
    "1. Select Tutorial-RAG\\tutorial-rag-requirements.txt for the dependencies.\n",
    "\n",
    "It takes several minutes to create the environment. When the environment is ready, continue to the next step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install -r tutorial-rag-requirements.txt --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set endpoints\n",
    "\n",
    "Provide the endpoints you collected in a previous step. You can leave the API keys empty if you enabled role-based authentication. Otherwise, if you can't use roles, provide API keys for each resource.\n",
    "\n",
    "The Azure AI multiservice account is used for skills processing. The multiservice account key must be provided, even if you're using role-based access control. The key isn't used on the connection, but it's currently used for billing purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set endpoints and API keys for Azure services\n",
    "AZURE_SEARCH_SERVICE: str = \"PUT YOUR SEARCH SERVICE URL HERE\"\n",
    "# AZURE_SEARCH_KEY: str = \"DELETE IF USING ROLES, OTHERWISE PUT YOUR SEARCH SERVICE ADMIN KEY HERE\"\n",
    "AZURE_OPENAI_ACCOUNT: str = \"PUT YOUR AZURE OPENAI ACCOUNT URL HERE\"\n",
    "# AZURE_OPENAI_KEY: str = \"DELETE IF USING ROLES, OTHERWISE PUT YOUR AZURE OPENAI KEY HERE\"\n",
    "AZURE_AI_MULTISERVICE_ACCOUNT: str = \"PUT YOUR AZURE AI MULTISERVICE ACCOUNT URL HERE\"\n",
    "AZURE_AI_MULTISERVICE_KEY: str = \"PUT YOUR AZURE AI MULTISERVICE KEY HERE. ROLES ARE USED TO CONNECT. KEY IS USED FOR BILLING.\"\n",
    "AZURE_STORAGE_CONNECTION: str = \"PUT YOUR AZURE STORAGE CONNECTION STRING HERE (see example below for syntax)\"\n",
    "\n",
    "# Example connection string for a search service managed identity connection:\n",
    "# \"ResourceId=/subscriptions/FAKE-SUBCRIPTION=ID/resourceGroups/FAKE-RESOURCE-GROUP/providers/Microsoft.Storage/storageAccounts/FAKE-ACCOUNT;\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create an index\n",
    "\n",
    "This is index schema used for [Build a RAG solution in Azure AI Search](https://learn.microsoft.com/azure/search/tutorial-rag-build-solution)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "py-rag-tutorial-idx created\n"
     ]
    }
   ],
   "source": [
    "from azure.identity import DefaultAzureCredential\n",
    "from azure.identity import get_bearer_token_provider\n",
    "from azure.search.documents.indexes import SearchIndexClient\n",
    "from azure.search.documents.indexes.models import (\n",
    "    SearchField,\n",
    "    SearchFieldDataType,\n",
    "    VectorSearch,\n",
    "    HnswAlgorithmConfiguration,\n",
    "    VectorSearchProfile,\n",
    "    AzureOpenAIVectorizer,\n",
    "    AzureOpenAIVectorizerParameters,\n",
    "    SearchIndex\n",
    ")\n",
    "\n",
    "credential = DefaultAzureCredential()\n",
    "\n",
    "# Create a search index  \n",
    "index_name = \"py-rag-tutorial-idx\"\n",
    "index_client = SearchIndexClient(endpoint=AZURE_SEARCH_SERVICE, credential=credential)  \n",
    "fields = [\n",
    "    SearchField(name=\"parent_id\", type=SearchFieldDataType.String),  \n",
    "    SearchField(name=\"title\", type=SearchFieldDataType.String),\n",
    "    SearchField(name=\"locations\", type=SearchFieldDataType.Collection(SearchFieldDataType.String), filterable=True),\n",
    "    SearchField(name=\"chunk_id\", type=SearchFieldDataType.String, key=True, sortable=True, filterable=True, facetable=True, analyzer_name=\"keyword\"),  \n",
    "    SearchField(name=\"chunk\", type=SearchFieldDataType.String, sortable=False, filterable=False, facetable=False),  \n",
    "    SearchField(name=\"text_vector\", type=SearchFieldDataType.Collection(SearchFieldDataType.Single), vector_search_dimensions=1024, vector_search_profile_name=\"myHnswProfile\")\n",
    "    ]  \n",
    "  \n",
    "# Configure the vector search configuration  \n",
    "vector_search = VectorSearch(  \n",
    "    algorithms=[  \n",
    "        HnswAlgorithmConfiguration(name=\"myHnsw\"),\n",
    "    ],  \n",
    "    profiles=[  \n",
    "        VectorSearchProfile(  \n",
    "            name=\"myHnswProfile\",  \n",
    "            algorithm_configuration_name=\"myHnsw\",  \n",
    "            vectorizer_name=\"myOpenAI\",  \n",
    "        )\n",
    "    ],  \n",
    "    vectorizers=[  \n",
    "        AzureOpenAIVectorizer(  \n",
    "            vectorizer_name=\"myOpenAI\",  \n",
    "            kind=\"azureOpenAI\",  \n",
    "            parameters=AzureOpenAIVectorizerParameters(  \n",
    "                resource_url=AZURE_OPENAI_ACCOUNT,  \n",
    "                deployment_name=\"text-embedding-3-large\",\n",
    "                model_name=\"text-embedding-3-large\"\n",
    "            ),\n",
    "        ),  \n",
    "    ], \n",
    ")  \n",
    "  \n",
    "# Create the search index\n",
    "index = SearchIndex(name=index_name, fields=fields, vector_search=vector_search)  \n",
    "result = index_client.create_or_update_index(index)  \n",
    "print(f\"{result.name} created\")  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a data source\n",
    "\n",
    "Create a data source connection to PDFs in Azure Blob Storage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data source 'py-rag-tutorial-ds' created or updated\n"
     ]
    }
   ],
   "source": [
    "from azure.search.documents.indexes import SearchIndexerClient\n",
    "from azure.search.documents.indexes.models import (\n",
    "    SearchIndexerDataContainer,\n",
    "    SearchIndexerDataSourceConnection\n",
    ")\n",
    "\n",
    "# Create a data source \n",
    "indexer_client = SearchIndexerClient(endpoint=AZURE_SEARCH_SERVICE, credential=credential)\n",
    "container = SearchIndexerDataContainer(name=\"nasa-ebooks-pdfs-all\")\n",
    "data_source_connection = SearchIndexerDataSourceConnection(\n",
    "    name=\"py-rag-tutorial-ds\",\n",
    "    type=\"azureblob\",\n",
    "    connection_string=AZURE_STORAGE_CONNECTION,\n",
    "    container=container\n",
    ")\n",
    "data_source = indexer_client.create_or_update_data_source_connection(data_source_connection)\n",
    "\n",
    "print(f\"Data source '{data_source.name}' created or updated\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a skillset\n",
    "\n",
    "This skillset chunks and embeds data. It also uses entity recognition to detect location entities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "py-rag-tutorial-ss created\n"
     ]
    }
   ],
   "source": [
    "from azure.search.documents.indexes.models import (\n",
    "    SplitSkill,\n",
    "    InputFieldMappingEntry,\n",
    "    OutputFieldMappingEntry,\n",
    "    AzureOpenAIEmbeddingSkill,\n",
    "    EntityRecognitionSkill,\n",
    "    SearchIndexerIndexProjection,\n",
    "    SearchIndexerIndexProjectionSelector,\n",
    "    SearchIndexerIndexProjectionsParameters,\n",
    "    IndexProjectionMode,\n",
    "    SearchIndexerSkillset,\n",
    "    CognitiveServicesAccountKey\n",
    ")\n",
    "\n",
    "# Create a skillset  \n",
    "skillset_name = \"py-rag-tutorial-ss\"\n",
    "\n",
    "split_skill = SplitSkill(  \n",
    "    description=\"Split skill to chunk documents\",  \n",
    "    text_split_mode=\"pages\",  \n",
    "    context=\"/document\",  \n",
    "    maximum_page_length=2000,  \n",
    "    page_overlap_length=500,  \n",
    "    inputs=[  \n",
    "        InputFieldMappingEntry(name=\"text\", source=\"/document/content\"),  \n",
    "    ],  \n",
    "    outputs=[  \n",
    "        OutputFieldMappingEntry(name=\"textItems\", target_name=\"pages\")  \n",
    "    ],  \n",
    ")  \n",
    "  \n",
    "embedding_skill = AzureOpenAIEmbeddingSkill(  \n",
    "    description=\"Skill to generate embeddings via Azure OpenAI\",  \n",
    "    context=\"/document/pages/*\",  \n",
    "    resource_url=AZURE_OPENAI_ACCOUNT,  \n",
    "    deployment_name=\"text-embedding-3-large\",  \n",
    "    model_name=\"text-embedding-3-large\",\n",
    "    dimensions=1024,\n",
    "    inputs=[  \n",
    "        InputFieldMappingEntry(name=\"text\", source=\"/document/pages/*\"),  \n",
    "    ],  \n",
    "    outputs=[  \n",
    "        OutputFieldMappingEntry(name=\"embedding\", target_name=\"text_vector\")  \n",
    "    ],  \n",
    ")\n",
    "\n",
    "entity_skill = EntityRecognitionSkill(\n",
    "    description=\"Skill to recognize entities in text\",\n",
    "    context=\"/document/pages/*\",\n",
    "    categories=[\"Location\"],\n",
    "    default_language_code=\"en\",\n",
    "    inputs=[\n",
    "        InputFieldMappingEntry(name=\"text\", source=\"/document/pages/*\")\n",
    "    ],\n",
    "    outputs=[\n",
    "        OutputFieldMappingEntry(name=\"locations\", target_name=\"locations\")\n",
    "    ]\n",
    ")\n",
    "  \n",
    "index_projections = SearchIndexerIndexProjection(  \n",
    "    selectors=[  \n",
    "        SearchIndexerIndexProjectionSelector(  \n",
    "            target_index_name=index_name,  \n",
    "            parent_key_field_name=\"parent_id\",  \n",
    "            source_context=\"/document/pages/*\",  \n",
    "            mappings=[  \n",
    "                InputFieldMappingEntry(name=\"chunk\", source=\"/document/pages/*\"),  \n",
    "                InputFieldMappingEntry(name=\"text_vector\", source=\"/document/pages/*/text_vector\"),\n",
    "                InputFieldMappingEntry(name=\"locations\", source=\"/document/pages/*/locations\"),  \n",
    "                InputFieldMappingEntry(name=\"title\", source=\"/document/metadata_storage_name\"),  \n",
    "            ],  \n",
    "        ),  \n",
    "    ],  \n",
    "    parameters=SearchIndexerIndexProjectionsParameters(  \n",
    "        projection_mode=IndexProjectionMode.SKIP_INDEXING_PARENT_DOCUMENTS  \n",
    "    ),  \n",
    ") \n",
    "\n",
    "cognitive_services_account = CognitiveServicesAccountKey(key=AZURE_AI_MULTISERVICE_KEY)\n",
    "\n",
    "skills = [split_skill, embedding_skill, entity_skill]\n",
    "\n",
    "skillset = SearchIndexerSkillset(  \n",
    "    name=skillset_name,  \n",
    "    description=\"Skillset to chunk documents and generating embeddings\",  \n",
    "    skills=skills,  \n",
    "    index_projection=index_projections,\n",
    "    cognitive_services_account=cognitive_services_account\n",
    ")\n",
    "  \n",
    "client = SearchIndexerClient(endpoint=AZURE_SEARCH_SERVICE, credential=credential)  \n",
    "client.create_or_update_skillset(skillset)  \n",
    "print(f\"{skillset.name} created\")  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create an indexer\n",
    "\n",
    "The indexer drives the pipeline. You can create an indexer in a disabled state, but the default is for the indexer to run as soon as you send the request."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " py-rag-tutorial-idxr is created and running. Give the indexer a few minutes before running a query.\n"
     ]
    }
   ],
   "source": [
    "from azure.search.documents.indexes.models import (\n",
    "    SearchIndexer\n",
    ")\n",
    "\n",
    "# Create an indexer  \n",
    "indexer_name = \"py-rag-tutorial-idxr\" \n",
    "\n",
    "indexer_parameters = None\n",
    "\n",
    "indexer = SearchIndexer(  \n",
    "    name=indexer_name,  \n",
    "    description=\"Indexer to index documents and generate embeddings\",  \n",
    "    skillset_name=skillset_name,  \n",
    "    target_index_name=index_name,  \n",
    "    data_source_name=data_source.name,\n",
    "    parameters=indexer_parameters\n",
    ")  \n",
    "\n",
    "# Create and run the indexer  \n",
    "indexer_client = SearchIndexerClient(endpoint=AZURE_SEARCH_SERVICE, credential=credential)  \n",
    "indexer_result = indexer_client.create_or_update_indexer(indexer)  \n",
    "\n",
    "print(f' {indexer_name} is created and running. Give the indexer a few minutes before running a query.')  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check results\n",
    "\n",
    "After waiting several minutes, send a request to the search engine. There is no chat or generative AI at this point. The results are verbatim content from your search index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score: 0.01666666753590107\n",
      "Chunk: national Aeronautics and Space Administration\n",
      "\n",
      "earth Science\n",
      "\n",
      "NASA Headquarters \n",
      "\n",
      "300 E Street SW \n",
      "\n",
      "Washington, DC 20546\n",
      "\n",
      "www.nasa.gov\n",
      "\n",
      "np-2018-05-2546-hQ\n"
     ]
    }
   ],
   "source": [
    "from azure.search.documents import SearchClient\n",
    "from azure.search.documents.models import VectorizableTextQuery\n",
    "\n",
    "# Vector Search using text-to-vector conversion of the query string\n",
    "query = \"what's NASA's website?\"  \n",
    "\n",
    "search_client = SearchClient(endpoint=AZURE_SEARCH_SERVICE, credential=credential, index_name=index_name)\n",
    "vector_query = VectorizableTextQuery(text=query, k_nearest_neighbors=50, fields=\"text_vector\")\n",
    "  \n",
    "results = search_client.search(  \n",
    "    search_text=query,  \n",
    "    vector_queries= [vector_query],\n",
    "    select=[\"chunk\"],\n",
    "    top=1\n",
    ")  \n",
    "  \n",
    "for result in results:  \n",
    "    print(f\"Score: {result['@search.score']}\")\n",
    "    print(f\"Chunk: {result['chunk']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Search using a chat model\n",
    "\n",
    "This script sends a query, the query response, and a prompt to an LLM for chat completion. This time, the response is created using generative AI. We broke this task out into three separate tasks: set up the clients, set up the search query, call the LLM and get the response. We also give it a more interesting query. \n",
    "\n",
    "To learn more about this step, revisit [Search and generate answers](https://learn.microsoft.com/azure/search/tutorial-rag-build-solution-query) in the tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import libraries\n",
    "from azure.search.documents import SearchClient\n",
    "from openai import AzureOpenAI\n",
    "\n",
    "# Set up the Azure OpenAI client\n",
    "token_provider = get_bearer_token_provider(credential, \"https://cognitiveservices.azure.com/.default\")\n",
    "openai_client = AzureOpenAI(\n",
    "     api_version=\"2024-06-01\",\n",
    "     azure_endpoint=AZURE_OPENAI_ACCOUNT,\n",
    "     azure_ad_token_provider=token_provider\n",
    " )\n",
    "\n",
    "deployment_name = \"gpt-4o\"\n",
    "\n",
    "# Set up the Azure Azure AI Search client\n",
    "search_client = SearchClient(\n",
    "     endpoint=AZURE_SEARCH_SERVICE,\n",
    "     index_name=index_name,\n",
    "     credential=credential\n",
    " )\n",
    "\n",
    "# Provide instructions to the model\n",
    "GROUNDED_PROMPT=\"\"\"\n",
    "You are an AI assistant that helps users learn from the information found in the source material.\n",
    "Answer the query using only the sources provided below.\n",
    "Use bullets if the answer has multiple points.\n",
    "If the answer is longer than 3 sentences, provide a summary.\n",
    "Answer ONLY with the facts listed in the list of sources below. Cite your source when you answer the question\n",
    "If there isn't enough information below, say you don't know.\n",
    "Do not generate answers that don't use the sources below.\n",
    "Query: {query}\n",
    "Sources:\\n{sources}\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Provide the search query. \n",
    "# It's hybrid: a keyword search on \"query\", with text-to-vector conversion for \"vector_query\".\n",
    "# The vector query finds 50 nearest neighbor matches in the search index\n",
    "query=\"What's the NASA earth book about?\"\n",
    "vector_query = VectorizableTextQuery(text=query, k_nearest_neighbors=50, fields=\"text_vector\")\n",
    "\n",
    "# Set up the search results and the chat thread.\n",
    "# Retrieve the selected fields from the search index related to the question.\n",
    "# Search results are limited to the top 5 matches. Limiting top can help you stay under LLM quotas.\n",
    "search_results = search_client.search(\n",
    "    search_text=query,\n",
    "    vector_queries= [vector_query],\n",
    "    select=[\"title\", \"chunk\", \"locations\"],\n",
    "    top=5,\n",
    ")\n",
    "\n",
    "# Newlines could be in the OCR'd content or in PDFs, as is the case for the sample PDFs used for this tutorial.\n",
    "# Use a unique separator to make the sources distinct. \n",
    "# We chose repeated equal signs (=) followed by a newline because it's unlikely the source documents contain this sequence.\n",
    "sources_formatted = \"=================\\n\".join([f'TITLE: {document[\"title\"]}, CONTENT: {document[\"chunk\"]}, LOCATIONS: {document[\"locations\"]}' for document in search_results])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The NASA Earth book is about the intricate and captivating science of our planet, studied through NASA's unique perspective and tools. It presents Earth as a dynamic and complex system, observed through various cycles and processes such as the water cycle and ocean circulation. The book combines stunning satellite images with detailed scientific insights, portraying Earth’s beauty and the continuous interaction of land, wind, water, ice, and air seen from above. It aims to inspire and demonstrate that the truth of our planet is as compelling as any fiction.\n",
      "\n",
      "Source: page-8.pdf\n"
     ]
    }
   ],
   "source": [
    "response = openai_client.chat.completions.create(\n",
    "    messages=[\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": GROUNDED_PROMPT.format(query=query, sources=sources_formatted)\n",
    "        }\n",
    "    ],\n",
    "    model=deployment_name\n",
    ")\n",
    "\n",
    "print(response.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Try another query\n",
    "\n",
    "The first query is very broad. Let's ask another question that requires the search engine and the LLM to find more granular information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Yes, there are cloud formations specific to oceans and large bodies of water. These include:\n",
      "\n",
      "- **Cloud Streets**: Formed when wind blows from a cold surface like sea ice over the warmer, moister air near the open ocean. The winds create cylinders of spinning air, with clouds forming along the upward cycle of these cylinders. This phenomenon was observed over the Bering Strait in January 2010 (Source: page-21.pdf).\n",
      "- **Dense Marine Clouds**: Commonly form over the ocean due to cooler, moist marine air. For example, along the coast of China, onshore winds carry these clouds toward the land, but they tend to evaporate as they move onshore due to the warmer, drier landmass (Source: page-33.pdf).\n",
      "\n",
      "Summary: Specific cloud formations, such as cloud streets and dense marine clouds, occur over oceans and large bodies of water.\n"
     ]
    }
   ],
   "source": [
    "# Focused query on cloud formations and bodies of water\n",
    "query=\"Are there any cloud formations specific to oceans and large bodies of water?\"\n",
    "vector_query = VectorizableTextQuery(text=query, k_nearest_neighbors=50, fields=\"text_vector\")\n",
    "\n",
    "search_results = search_client.search(\n",
    "    search_text=query,\n",
    "    vector_queries= [vector_query],\n",
    "    select=[\"title\", \"chunk\", \"locations\"],\n",
    "    top=5,\n",
    ")\n",
    "\n",
    "sources_formatted = \"=================\\n\".join([f'TITLE: {document[\"title\"]}, CONTENT: {document[\"chunk\"]}, LOCATIONS: {document[\"locations\"]}' for document in search_results])\n",
    "\n",
    "response = openai_client.chat.completions.create(\n",
    "    messages=[\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": GROUNDED_PROMPT.format(query=query, sources=sources_formatted)\n",
    "        }\n",
    "    ],\n",
    "    model=deployment_name\n",
    ")\n",
    "\n",
    "print(response.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Update the schema for semantic ranking and scoring profile\n",
    "\n",
    "Azure AI Search has multiple features and capabilities that improve relevance. In this step, we add two of them: semantic ranking and scoring profiles. \n",
    "\n",
    "Semantic ranking and scoring profile configurations exist in the index schema. You can update an existing index to use both without incurring a [rebuild requirement](/azure/search/search-howto-reindex). An update request should include all of the existing schema definitions that you want to keep, plus the new or changed elements. It's a best practice to issue a GET INDEX request to retrieve the current index before adding new elements.\n",
    "\n",
    "To learn more about this step, see [Maximimze relevance](https://learn.microsoft.com/azure/search/tutorial-rag-build-maximize-relevance) in the RAG tutorial series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "py-rag-tutorial-idx updated\n"
     ]
    }
   ],
   "source": [
    "# Update the classes to include the new fields\n",
    "from azure.identity import DefaultAzureCredential\n",
    "from azure.identity import get_bearer_token_provider\n",
    "from azure.search.documents.indexes import SearchIndexClient\n",
    "from azure.search.documents.indexes.models import (\n",
    "    SearchField,\n",
    "    SearchFieldDataType,\n",
    "    VectorSearch,\n",
    "    HnswAlgorithmConfiguration,\n",
    "    VectorSearchProfile,\n",
    "    AzureOpenAIVectorizer,\n",
    "    AzureOpenAIVectorizerParameters,\n",
    "    SearchIndex,\n",
    "    SemanticConfiguration,\n",
    "    SemanticPrioritizedFields,\n",
    "    SemanticField,\n",
    "    SemanticSearch,\n",
    "    ScoringProfile,\n",
    "    TagScoringFunction,\n",
    "    TagScoringParameters\n",
    ")\n",
    "\n",
    "credential = DefaultAzureCredential()\n",
    "\n",
    "# Existing index name, client and fields\n",
    "index_name = \"py-rag-tutorial-idx\"\n",
    "index_client = SearchIndexClient(endpoint=AZURE_SEARCH_SERVICE, credential=credential)  \n",
    "fields = [\n",
    "    SearchField(name=\"parent_id\", type=SearchFieldDataType.String),  \n",
    "    SearchField(name=\"title\", type=SearchFieldDataType.String),\n",
    "    SearchField(name=\"locations\", type=SearchFieldDataType.Collection(SearchFieldDataType.String), filterable=True),\n",
    "    SearchField(name=\"chunk_id\", type=SearchFieldDataType.String, key=True, sortable=True, filterable=True, facetable=True, analyzer_name=\"keyword\"),  \n",
    "    SearchField(name=\"chunk\", type=SearchFieldDataType.String, sortable=False, filterable=False, facetable=False),  \n",
    "    SearchField(name=\"text_vector\", type=SearchFieldDataType.Collection(SearchFieldDataType.Single), vector_search_dimensions=1024, vector_search_profile_name=\"myHnswProfile\")\n",
    "    ]  \n",
    "  \n",
    "# Existing vector search configuration  \n",
    "vector_search = VectorSearch(  \n",
    "    algorithms=[  \n",
    "        HnswAlgorithmConfiguration(name=\"myHnsw\"),\n",
    "    ],  \n",
    "    profiles=[  \n",
    "        VectorSearchProfile(  \n",
    "            name=\"myHnswProfile\",  \n",
    "            algorithm_configuration_name=\"myHnsw\",  \n",
    "            vectorizer_name=\"myOpenAI\",  \n",
    "        )\n",
    "    ],  \n",
    "    vectorizers=[  \n",
    "        AzureOpenAIVectorizer(  \n",
    "            vectorizer_name=\"myOpenAI\",  \n",
    "            kind=\"azureOpenAI\",  \n",
    "            parameters=AzureOpenAIVectorizerParameters(  \n",
    "                resource_url=AZURE_OPENAI_ACCOUNT,  \n",
    "                deployment_name=\"text-embedding-3-large\",\n",
    "                model_name=\"text-embedding-3-large\"\n",
    "            ),\n",
    "        ),  \n",
    "    ], \n",
    ")\n",
    "\n",
    "# New semantic configuration\n",
    "semantic_config = SemanticConfiguration(\n",
    "    name=\"my-semantic-config\",\n",
    "    prioritized_fields=SemanticPrioritizedFields(\n",
    "        title_field=SemanticField(field_name=\"title\"),\n",
    "        keywords_fields=[SemanticField(field_name=\"locations\")],\n",
    "        content_fields=[SemanticField(field_name=\"chunk\")]\n",
    "    )\n",
    ")\n",
    "\n",
    "# Create the semantic settings with the configuration\n",
    "semantic_search = SemanticSearch(configurations=[semantic_config])\n",
    "\n",
    "# New scoring profile\n",
    "scoring_profiles = [  \n",
    "    ScoringProfile(  \n",
    "        name=\"my-scoring-profile\",\n",
    "        functions=[\n",
    "            TagScoringFunction(  \n",
    "                field_name=\"locations\",  \n",
    "                boost=5.0,  \n",
    "                parameters=TagScoringParameters(  \n",
    "                    tags_parameter=\"tags\",  \n",
    "                ),  \n",
    "            ) \n",
    "        ]\n",
    "    )\n",
    "]\n",
    "\n",
    "\n",
    "# Update the search index with the semantic configuration and scoring profile\n",
    "index = SearchIndex(name=index_name, fields=fields, vector_search=vector_search, semantic_search=semantic_search, scoring_profiles=scoring_profiles)  \n",
    "result = index_client.create_or_update_index(index)  \n",
    "print(f\"{result.name} updated\")  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Rerun the query using semantic configuration and scoring profile\n",
    "\n",
    "This example updates the query request. It's using the same query as before, but with the addition of semantic ranking and a scoring profile that boosts any matching search documents that mention water-related terms. \n",
    "\n",
    "Compared to the \"before\" query that gave us a reasonable response to the question about cloud formations and water, this query should provide a better answer based on the extra relevance tuning capabilities.\n",
    "\n",
    "An explanation for this script can be found in [Maximimze relevance](https://learn.microsoft.com/azure/search/tutorial-rag-build-maximize-relevance) in the RAG tutorial series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Yes, there are specific cloud formations associated with oceans and large bodies of water:\n",
      "\n",
      "- **Low Stratus Clouds**: Observed framing a hole over iceberg A-56 in the South Atlantic Ocean. These clouds can be influenced by thermal instabilities created by large obstacles like icebergs (page-39.pdf).\n",
      "- **Undular Bore/Solitary Wave**: Created by the interaction between cool, dry air from Africa and warm, moist air over the Atlantic Ocean off the coast of Mauritania. This results in a wave structure in the atmosphere that influences cloud formation (page-23.pdf).\n",
      "- **Ship Tracks**: Narrow clouds formed by water vapor condensing around pollution particles from ship exhaust, observed over the Pacific Ocean. These clouds can stretch for many hundreds of kilometers (page-31.pdf).\n",
      "- **Volcanic Eruption Plumes**: Ash and volcanic particles from eruptions in the South Sandwich Islands act as seeds for cloud formation. These plumes were observed in the South Atlantic Ocean (page-13.pdf).\n",
      "\n",
      "Summary: Specific cloud formations over oceans include low stratus clouds, undular bores, ship tracks, and volcanic eruption plumes, influenced by various factors such as thermal instabilities, air interactions, pollution, and volcanic activity.\n"
     ]
    }
   ],
   "source": [
    "# Import libraries\n",
    "from azure.search.documents import SearchClient\n",
    "from openai import AzureOpenAI\n",
    "\n",
    "token_provider = get_bearer_token_provider(credential, \"https://cognitiveservices.azure.com/.default\")\n",
    "openai_client = AzureOpenAI(\n",
    "     api_version=\"2024-06-01\",\n",
    "     azure_endpoint=AZURE_OPENAI_ACCOUNT,\n",
    "     azure_ad_token_provider=token_provider\n",
    " )\n",
    "\n",
    "deployment_name = \"gpt-4o\"\n",
    "\n",
    "search_client = SearchClient(\n",
    "     endpoint=AZURE_SEARCH_SERVICE,\n",
    "     index_name=index_name,\n",
    "     credential=credential\n",
    " )\n",
    "\n",
    "# Prompt is unchanged in this update\n",
    "GROUNDED_PROMPT=\"\"\"\n",
    "You are an AI assistant that helps users learn from the information found in the source material.\n",
    "Answer the query using only the sources provided below.\n",
    "Use bullets if the answer has multiple points.\n",
    "If the answer is longer than 3 sentences, provide a summary.\n",
    "Answer ONLY with the facts listed in the list of sources below. Cite your source when you answer the question\n",
    "If there isn't enough information below, say you don't know.\n",
    "Do not generate answers that don't use the sources below.\n",
    "Query: {query}\n",
    "Sources:\\n{sources}\n",
    "\"\"\"\n",
    "\n",
    "# Queries are unchanged in this update\n",
    "query=\"Are there any cloud formations specific to oceans and large bodies of water?\"\n",
    "vector_query = VectorizableTextQuery(text=query, k_nearest_neighbors=50, fields=\"text_vector\")\n",
    "\n",
    "# Add query_type semantic and semantic_configuration_name\n",
    "# Add scoring_profile and scoring_parameters\n",
    "search_results = search_client.search(\n",
    "    query_type=\"semantic\",\n",
    "    semantic_configuration_name=\"my-semantic-config\",\n",
    "    scoring_profile=\"my-scoring-profile\",\n",
    "    scoring_parameters=[\"tags-ocean, 'sea surface', seas, surface\"],\n",
    "    search_text=query,\n",
    "    vector_queries= [vector_query],\n",
    "    select=\"title, chunk, locations\",\n",
    "    top=5,\n",
    ")\n",
    "sources_formatted = \"=================\\n\".join([f'TITLE: {document[\"title\"]}, CONTENT: {document[\"chunk\"]}, LOCATIONS: {document[\"locations\"]}' for document in search_results])\n",
    "\n",
    "response = openai_client.chat.completions.create(\n",
    "    messages=[\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": GROUNDED_PROMPT.format(query=query, sources=sources_formatted)\n",
    "        }\n",
    "    ],\n",
    "    model=deployment_name\n",
    ")\n",
    "\n",
    "print(response.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a second index with reduced vector size\n",
    "\n",
    "Azure AI Search has multiple approaches for reducing vector size, which lowers the cost of vector workloads. In this step, create a new index that uses the following capabilities:\n",
    "\n",
    "- Smaller vector indexes by compressing the vectors used during query execution. Scalar quantization provides this capability.\n",
    "- Smaller vector indexes by opting out of vector storage for search results. If you only need vectors for queries and not in response payload, you can drop the vector copy used for search results.\n",
    "- Smaller vector fields through narrow data types. You can specify Collection(Edm.Half) on the text_vector field to store incoming float32 dimensions as float16.\n",
    "\n",
    "All of these capabilities are specified in a search index. After you load the index, compare the difference between the original index and the new one.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.identity import DefaultAzureCredential\n",
    "from azure.identity import get_bearer_token_provider\n",
    "from azure.search.documents.indexes import SearchIndexClient\n",
    "from azure.search.documents.indexes.models import (\n",
    "    SearchField,\n",
    "    SearchFieldDataType,\n",
    "    VectorSearch,\n",
    "    HnswAlgorithmConfiguration,\n",
    "    VectorSearchProfile,\n",
    "    AzureOpenAIVectorizer,\n",
    "    AzureOpenAIVectorizerParameters,\n",
    "    ScalarQuantizationCompression,\n",
    "    ScalarQuantizationParameters,\n",
    "    SearchIndex,\n",
    "    SemanticConfiguration,\n",
    "    SemanticPrioritizedFields,\n",
    "    SemanticField,\n",
    "    SemanticSearch,\n",
    "    ScoringProfile,\n",
    "    TagScoringFunction,\n",
    "    TagScoringParameters\n",
    ")\n",
    "\n",
    "credential = DefaultAzureCredential()\n",
    "\n",
    "index_name = \"py-rag-tutorial-small-vectors-idx\"\n",
    "index_client = SearchIndexClient(endpoint=AZURE_SEARCH_SERVICE, credential=credential)  \n",
    "fields = [\n",
    "    SearchField(name=\"parent_id\", type=SearchFieldDataType.String),  \n",
    "    SearchField(name=\"title\", type=SearchFieldDataType.String),\n",
    "    SearchField(name=\"locations\", type=SearchFieldDataType.Collection(SearchFieldDataType.String), filterable=True),\n",
    "    SearchField(name=\"chunk_id\", type=SearchFieldDataType.String, key=True, sortable=True, filterable=True, facetable=True, analyzer_name=\"keyword\"),  \n",
    "    SearchField(name=\"chunk\", type=SearchFieldDataType.String, sortable=False, filterable=False, facetable=False),  \n",
    "    SearchField(name=\"text_vector\", type=\"Collection(Edm.Half)\", vector_search_dimensions=1024, vector_search_profile_name=\"myHnswProfile\", stored= False)\n",
    "    ]  \n",
    "\n",
    "# Configure the vector search configuration  \n",
    "vector_search = VectorSearch(  \n",
    "    algorithms=[  \n",
    "        HnswAlgorithmConfiguration(name=\"myHnsw\"),\n",
    "    ],  \n",
    "    profiles=[  \n",
    "        VectorSearchProfile(  \n",
    "            name=\"myHnswProfile\",  \n",
    "            algorithm_configuration_name=\"myHnsw\",\n",
    "            compression_name=\"myScalarQuantization\",\n",
    "            vectorizer_name=\"myOpenAI\",  \n",
    "        )\n",
    "    ],  \n",
    "    vectorizers=[  \n",
    "        AzureOpenAIVectorizer(  \n",
    "            vectorizer_name=\"myOpenAI\",  \n",
    "            kind=\"azureOpenAI\",  \n",
    "            parameters=AzureOpenAIVectorizerParameters(  \n",
    "                resource_url=AZURE_OPENAI_ACCOUNT,  \n",
    "                deployment_name=\"text-embedding-3-large\",\n",
    "                model_name=\"text-embedding-3-large\"\n",
    "            ),\n",
    "        ),  \n",
    "    ],\n",
    "    compressions=[\n",
    "        ScalarQuantizationCompression(\n",
    "            compression_name=\"myScalarQuantization\",\n",
    "            rerank_with_original_vectors=True,\n",
    "            default_oversampling=10,\n",
    "            parameters=ScalarQuantizationParameters(quantized_data_type=\"int8\"),\n",
    "        )\n",
    "    ]\n",
    ")\n",
    "\n",
    "semantic_config = SemanticConfiguration(\n",
    "    name=\"my-semantic-config\",\n",
    "    prioritized_fields=SemanticPrioritizedFields(\n",
    "        title_field=SemanticField(field_name=\"title\"),\n",
    "        keywords_fields=[SemanticField(field_name=\"locations\")],\n",
    "        content_fields=[SemanticField(field_name=\"chunk\")]\n",
    "    )\n",
    ")\n",
    "\n",
    "semantic_search = SemanticSearch(configurations=[semantic_config])\n",
    "\n",
    "scoring_profiles = [  \n",
    "    ScoringProfile(  \n",
    "        name=\"my-scoring-profile\",\n",
    "        functions=[\n",
    "            TagScoringFunction(  \n",
    "                field_name=\"locations\",  \n",
    "                boost=5.0,  \n",
    "                parameters=TagScoringParameters(  \n",
    "                    tags_parameter=\"tags\",  \n",
    "                ),  \n",
    "            ) \n",
    "        ]\n",
    "    )\n",
    "]\n",
    "\n",
    "index = SearchIndex(name=index_name, fields=fields, vector_search=vector_search, semantic_search=semantic_search, scoring_profiles=scoring_profiles)  \n",
    "result = index_client.create_or_update_index(index)  \n",
    "print(f\"{result.name} created\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run the indexer to create the new index\n",
    "\n",
    "Your ability to create and populate the new index is predicated on reusing data structures created earlier in this tutorial. If you have run every cell in this notebook, you have an existing data source and skillset, but here we create a new indexer so that there's no history or caching to get in the way. The indexer uses the notebook's current values for skillset_name, target_index_name, and data_source_name. Because the skillset and data source haven't changed, the existing references are still valid. Because you just created a new index in the previous cell, the current target_index_name is now \"py-rag-tutorial-small-vectors-idx\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.search.documents.indexes.models import (\n",
    "    SearchIndexer\n",
    ")\n",
    "\n",
    "# Create an indexer  \n",
    "indexer_name = \"py-rag-tutorial-small-vectors-idxr\" \n",
    "\n",
    "indexer_parameters = None\n",
    "\n",
    "indexer = SearchIndexer(  \n",
    "    name=indexer_name,  \n",
    "    description=\"Indexer to index documents and generate embeddings\",  \n",
    "    skillset_name=skillset_name,  \n",
    "    target_index_name=index_name,  \n",
    "    data_source_name=data_source.name,\n",
    "    parameters=indexer_parameters\n",
    ")  \n",
    "\n",
    "# Create and run the indexer  \n",
    "indexer_client = SearchIndexerClient(endpoint=AZURE_SEARCH_SERVICE, credential=credential)  \n",
    "indexer_result = indexer_client.create_or_update_indexer(indexer)  \n",
    "\n",
    "print(f' {indexer_name} is created and running. Give the indexer a few minutes before running a query.')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a final step, switch to the Azure portal to compare the vector storage requirements for the two indexes. The index created in the last step uses half-precision floating-point numbers (float16) for the text vectors. This reduces the storage requirements for the vectors by half compared to the previous index that used single-precision floating-point numbers (float32). Scalar compression and the omission of one set of the vecctors account for the remaining storage savings. For more information about reducing vector size, see [Choose an approach for optimizing vector storage and processing](https://learn.microsoft.com/azure/search/vector-search-how-to-configure-compression-storage).\n",
    "\n",
    "Consider revisiting the queries from the previous lessons so that you can compare query speed and utility. You should expect some variation in LLM output whenever you repeat a query, but in general the storage-saving techniques you implemented should not cause significant degradation of search result quality."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
